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ABSTfi&CT 

The effectiveness of instructional develcpaent 
evaluation programs is assessed. It is suggested that although it is 
a basic tenet in instructional development that teachi!\g iaproveoent 
is closely linked to effective evaluation, it is ircnical that cost 
instructional develcpaent programs have tJreaselves been evaluated 
only superficially, if at all. There is very liiited evidence that 
teaching practices and learning effectiveness have been substantially 
changed as a result of th€ instructional develofsent. Evaluation 
strategies on three levels are discussed: (1) activity within the 
program, which can be aonitored ir terms of number of contacts aade 
euid distribution "f instructional materials; (2) attitudes can be 
measured (teaching, learning, and program) ; and (3) the collection of 
eiapirical evidence for changes related to improved learning and 
teaching. In practice most evaluation of instructional develcpaent 
programs has been conjoined to the first two levels. A recent inforaai 
survey of instructional' developers in several countries revealed that 
not only are evaluation efforts scarce, but many instructional 
developers are resistant tc the very notion o€ foriaal assessment of 
their activities. The reasons for this, such as budgetary 
considerations, are explored. (Author/PHB) 
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Tae past decade has seen an" exp5.osion in the nuiaber of formal units that 
have been established in universities to encourage the ituprovement, of teaching 
^ ' effectiveness. Although it is a basic tenet in instructional development t^at ^ 
teaching inprov^nt is closely linked to effecti.ve evaluation, it is ironical 
that cost instructional development prograraoes have thecsselves been evaluated 
only superficially, if at all. In particular, there is very scaj;ity evidence 
that teaching practices and learning effectiveness have been substantially 
changed as a result of the instructional development movement. A number of 
evaluation strategies are possible. At the most basic level, activity within 
the prograime can be monitored in terms of number of contacts made, distribu- 
tion of instructional materials, and so on; the second level involves the 
raasuresent of attitudes — both to teaching and learning and to the programme 
itself; the third level requires the collection of empirical evidence for 
changes related to improved learning and teaching. In practice, most evaluation 
cf instructional development programmes has been confined to the first two levels- 
A recent informal survey of Instructional developers in several countries reveals 
that not only are evaluation efforts sparse, but that many instructional 
developers are resistant to the very notion of formal assessment of their I 
activities. The reasons for this state of affairs are explored. 

MODELS FOR PROGRAMME EVALUATION 
.fS The recent prolific growth of instructional development programmes in North 

Ai:erican universities has been documented elsewhere (e.g. Wergin, 1977). Since 
the 1960s such programmes have developed and expanded not only in the United 
^ S«btes and Canada, but also in Britain, Australia, New Zealand and many coun- 
tries of continential Europe — as evidenced by the wide geographical distribtftion 
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of contributors to the present set of proceedings- Despite the existence bf 
different tsodels of instructional development in different locations (se^ 

m 

O'Conaell «nd ^leeth. 1978, pp. 11-12, for a usefia categorization), a conaon 
eleJaent in the vork of nearly all prograsaaes has been an eophasis upon the 
importance of evaluation of instruction. Hence it is paradoxical that the 
assessment of instructional development prograjanes themselves is a matter of 
comparatively recent concern. For exaa^le, only three years ago. Rose (1976) 
felt it necessary to argue vehemAitly in favour of what she called "holistic 
evaluation" of professional development pi;;ograjBmes, though unfortunately she 
-^.^^ clues as to how to go about this task. Llr.dquist (1978) is another 
who has advocated systematic programme evaluation by means of a variety of 
methods. He lists evaluative criteria that inblude changes in faculty 
behaviour and attitudes as well as improvement in student learning, if possible 
measured across institutions. 

Among those offering specific evaluative prescriptions in this area arc ^ 
Abedor and Gustafson (1971), Bergquist and Phillips (1977), O'Connell and 
Meeth (1978) and Wergin (1977) . The last author, following Stuff lebeam et al 
(1971) distinguishes between the evaluation of proceW and products, and Abedor 
and Gustafson (1971) have provided a set of criteria Velaced to each. They 
continue by taking up a conmon theme: the distiactxon\between evaluation in 
terms of "measurable effects" as opposed to the "opinioiii? of proponents" 
(p» 2i)» and later discuss the difference between short term effects and more 
lasting long term gains. Wergin* s paper includes a helpful discussion of the 
classical scientific approaches to programme evaluation, indicating the prac- 
tical limitations of experimental designs when applied to most instructional 
development programmes. More recently, Bergquist and Phillips (1977) have 
raised the important question of whether evaluation should be focussed primarily 
on changing faculty attitudes and "faculty growth" or upon enhanced student 
learning. They identify eight steps to effective programme assessment. 
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PROGRAMME EVALUATION lU PRACTICE 
So such for the principles of progracnM evaluation. However, an- inter- 
esting corollary question is the extent to which these principles are put into 
practice in real life instructional development. In an atteinpt to provide a 
partial answer to this question, the author wrote to the directors of tea 
established and fairly prossinent instructional development progratames in 
Australia, Denmark, England, New Zealand, and the United States. Probably 
because nearly a3J. the people approached wete personally known to the writer, 
all of theo replied to the request for informatiou about their evaluatioa 
philosophy and practice. In some cases "the correspondence became qtdte 
lengthy, and in many instances useful additional documentation was provided* 
Although the san^le is obviously very small and subjectively selected, the 
cements were extremely frank and informative — possibly because the topic 
appeared to be an especially salient one for the respondents concerned. . 
Thus their replies formed the bas^^ of an ipteresting — if incomplete — 
cross-section of philosophies and practices of programme evaluation in five 
different national contexts* 

Table 1 lists the types of evaluation activities that the different 
centres tised in connection with their programmes, classified in terms of the 
three levels described above. Obviously the most striking aspect of the table 
is the paucity of activities at Level 3, despite the frequent comments in the 
evaluation literature about the importance of measuring "outcomes", "real 
change", and so on- It is interesting that the two programmes that show the 
-Dost Level 3 evaluation activity are both from the USA (where perhaps the 
cal]^ for educational accountability have been surillest) , both are regarded 
as extrariely successful prograranes, and — not coincidentally — both have 
Sufficient staff to devote considerable energy to the programme evaluation 
process* 



beginaing with the ^'identificatiou of program goals, priorities and values'* 
arid ending with the appraisal of the evaluation process itself by both 
evaluator and client (p. 290). The same authors describe seven evaluation 
models that can be applied in this field, encompassing both descriptive and 
experimental approaches. The different sorts of possible programsie outcomes 
are listed by O'Connell and Meeth (1978), who distinguish between effects on 
faculty, on students, on the administration, and upon the institution itself. 
These authors also list various types of relevant evidence that could be 
gathered to demonstrate such outcomes nave been achieved, including self*- 
reports, observations, examination of relevant records and reports, as well as 
eiipirical tests that indicate change has taken place. 

In an oversimplified form, the recommendations of these, and other, 
commentators on the programme evaluation process can be conceptualized in terms 
of appraisal at three different levels. The first, and simplest level, 
involves the relatively straightforward monitoring of activities generated 
by, and in response to, the instructional development programme. This type 
of evalua^.ton might include counting the number of requests for advice and 
assistance, number of newsletters distributed, number of books borrowed from 
the resource library, and so on. The second level of ap^^raisal involves 
the measurement of attitudes and perceptions, both in relation to the pro- 
gramme itself and to wider issues of teaching and learning. Opinions of 
faculty, students, administrators, and even the general public, all could 
have relevance here. The third level of evaluation requires the compilation 
of evidence for change. Most commentators tend to cite changes in behaviour 
in this connection, especially improvement in student learning, but clearly 
these are not the only changes of interest. For example, changed faculty 
attitudes and motivation might be equally important, as might a changed 
institutional climate, especially if this resulted in increased student 
enrollments. 



TABLE l! EVALPATIOM OF ISSTRUCIIOSjO, DEVEIOPHEST 



PROGRAMMES AT 10 INSTITimOSS 



LEVEL 1: 
Monitoring Activities 



Institution 3 (Australia) 
Nimber of fais^iilty seconded 
to programme; Requests for 
fixnds to support teaching 
innovations 

Institution 4 (Denmark) 
Renewed contacts with 
former programme partici- 
pants; Contacts from 
others in same department 
Institution 5 (England) 
Review of activities by 
external committee - 



Institution 7 (USA) 
Number of requests for 
services; "Repeat" cli'-- 
ents; Faculty adoptions 
of advice; Letters of 
praise ard thanks; Vi- 
sits by outsiders; Use 
of centre facilities 
Institution 8 (USA) 
Number atti^nding confer- 
ences and workshops, es- 
pecially from other . 
institutions ; Develop- 
ment of textbooks on 
basis of programme acti- 
vities; Attraction of 
outside funding 

' f — 

■* Nearly all programmes 
provide general descrip- 
tions of their activities 
in the form of annual 
reports. 



LEVEL 2: 
Measuring Attitudes 

Institution 1 (Australia) 
Informal feedback on value 
of ser^'ices 

Institution 2 (Australia) 
Follow-up survey of partici- 
pants in teiachlng courses 
Institution 3 (Australia) 
"Favourable public comment" 
on work of the imit 



Instituti6n 4 (Denmark) 
Questionnaire surveys and 
oral discussions with pro- 
gramme participants 



LEVEL 3: 
Demonstrating Change 



Institution 6 (New Zealand) 
(Planned) comparative study of 
self-perceptions of instruc- 
tional developers and percep- 
tions of others in institution 
(faculty, students, adminis- 
trators) 

Institution 7 (USA) 



Questionnaire surveys of 
publications and workshops; 
Interviews with clients by 
external evaluation team; 
(Planned) inters/lews with 
random sample of faculty 

Institution 8 (USA) 
Attitudes of persons secon-^ 
ded to programme; Favourable 
comments by students 



Institution 9 (USA) 
Attitudes of participating 
faculty; Comments from 
faculty and administrators 

Institution 10 (USA) 
"Awareness survey" of ran- 
dom sample of faculty; 
Attituues of grant recipi- 
ents 



Institution 8 (USA) 
Continuance of projects 
after central funding has 
ea^ed; Replication of ac^ 
tivities in othei insti- 
tutions (including use of 
centre publications) 



Institution 9 (0SA) 
Increase in student enroll- 
ment in redesigxxed courses; 
Changed student attitudes 
to courses; Administrative 
changes in course credit 
system 
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Of even laore interest, however, -are the comments provided by respondents 
indicating their basic philosophies of programme evaluation. Nearly all the 
directors comaented on the difficulty of the process and, perhaps surprisingly, 
a small majority were extremely hesitant — if not downright hostile ~ about 
the usefulness of conventional programme evaluation techniques as applied to 
instructional development. This attitude was particularly evident in programmes 
outside the Onited States « For example; the director of a programme in New 
Zealand expressed scepticism about using the criterion of "the nximbers of 
people making use of the resources" (Level 1 type evaluation) . He pointed out 
that some Australian cent res quote upwards of 90% of faculty making contacts 
with the instructional development service. "This would indicate a very 
effective Centre with extensive impact. However, if one looks closer, one 
' finds that the contact included those who even came to borrow a slide 
projector". In similar vein, the director of a very large Australian unit 
(presumably not the one referred to above) commented favourably about 
evaluations which are essentially "number crunching" activities • TRls res- 
pondent saw his unit as a change agent — "However, if I am doing my job 
perfectly, it may be I will suggest change to a Head of Department in such a 
way that he belie. veo that the idea is his". 

Much the same point about unrecognized catalytic effects is made elo- 
quently by another Australian director whose comments are worth quoting at 
some length. 

The assertion that thz luuU has been an excellent invest- 
irent for the University is impossible to prove, and in my view, 
it would be undesirable to attempt to prove... By the very 
nature of tht tuiLt^6 operations, our most significant achieve- 
ments are the least tangible ones. It is, for example, easy to 
point to the research we have done, easy to list academic staff 
who have funds for teaching innovations from us; it is difficult 
and often impossible to demonstrate how that research^and those 
teaching innovations have benefitted the University... 

The most significant point, which has a delicious irony, is 
this. Tkz Uiiit is most successful when its influence is not 
even recognized. When a seed is sown, a train of thought started, 
an interest stimulated, and later — perhaps much later — an 



* acadeoic teadies better, or his students iaam better or work 

better, that academic may have no inkling that the, \uuJt had 
any part in it* That is ideal. If he thinks he is entirely 
responsible for some change for the better, if it^ is genuinely 
his own, he persists in it, builds on it, is involved in it, 
much more effectively than if it comas from outside, from some- 
body else. It is iron.ic, it puts iuiit in a curious dilemma , 
* that to receive> no credit for something we have done is our 
highest achievement. 

Since this programme is apparently wej.1 regarded in its own university, has 
existed successfully for a number of years, and has recently 'received 
Increased financial support, it appears that this respondent has correctly 
judged the prevailing institutional climate, and that his approach to, evalu- 
ation is an acceptable one in that context. Judging by the contents of 
annual reports from ;^orth Americ^ instructional development programmes 
(partly reflected by the data tabiilatad in Table 1) , the environment in 
North American institutions is very different, and requires that at least 
lip service be paid to the importance of gathering tangible evidence of 
activities and effects — even though the quality of such evidence may often 
be questionable. 

SOME CONCLUDING COMENTS 
Analysis of the responses provided by instructional development programme 
directors from different institutional settings raises a number of general 
questions with regard to the evaluation process. Firstly, should the main 
purpose of the evaluation be to help programme staff improve their perfor- 
mance (formative evaluation) or to enable administrators to judge the 
programme's overall effectiveness at certain points in time (summative evalu- 
ation)? Linked with this is a second question of who should set the criteria 
for evaluation: programme staff? clientele? faculty at large? the adminis- 
tration? students? the wider public? Third is the important matter of how the 
evaluation should take account of the context in which the programme operates, 
including the institutional clisiate, and norms within the wider academic 
cotsmuiiity of that country. Fourth is the question of the intangible 



"catalytic" effects referred to by so many of the Australasian directors: 



one interesting exaiaple of such an effect is the fact that, since this informal 

t 

study began, the New Zealand respondent has embarked upon a project for the 
evalx^ation of faculty development programmes throughout that country, 
"stimulated to a certain extent by your original letter". Finally there is 
the important matter ^^jf the resources available to cotiduct programme evalua- 
tion. Many existing prograaaes have extremely modest budgets 'and sfifr, and 
to bring' in outside evaluators would be prohibitively esqiensive, while to 
devote staff tiioe to assessment would severely limit the time and* re£>ource« 
available for instructional development itself. This brings up the interesting 
evaluative technique of cost benefit analysis, which is widely used in 
industrial settings, but is rarely dealt with in the literature on evaluation 
of instructional^ development prograrones — though this approach might indeed 
provide a useful criterion to keep in mind when selecting evaluation, 
strategies for a modestly staffed unit operating in a large institution. 
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